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1.  INTRODUCTION 

Our  goal  is  to  revolutionize  brain-machine  interfaces.  Our  premise  is  that  this  can  be  done  by  developing 
advanced  software  that  allows  devices  to  communicate  far  more  efficiently  with  the  brain  than  ever  before, 
overcoming  major  limitations  of  present-day  physical  interfaces.  Specifically,  all  modern  brain-machine 
interfaces  (e.g.  deep  brain  stimulators,  cochlear  prostheses,  motor  implants)  have  resolution,  scale,  and 
architecture  that  are  poorly  matched  to  the  underlying  neural  circuitry,  a  situation  that  is  expected  to  persist  for 
decades.  Because  of  this  limitation,  current  interfaces  provide,  at  best,  crude  interactions  with  the  neural  signals 
that  govern  sensation,  action,  and  other  brain  functions.  We  aim  to  make  these  devices  far  more  useful  by 
encoding  signals  in  a  way  that  optimizes  the  information  transmitted  from  the  device  to  the  brain,  within  the  realm 
of  what  is  achievable  using  the  device.  This  optimal  achievable  encoding  approach  is  novel  and  unique.  We  tackle 
this  problem  for  a  particularly  promising  brain-machine  interface,  a  retinal  prosthesis.  In  this  application,  we 
leverage  our  understanding  of  retinal  circuitry  and  function,  along  with  modern  statistical  tools,  in  order  to 
develop  an  advanced  prosthesis  encoding  approach  that  most  accurately  mimics  naturalistic  retinal  function. 
Although  a  retinal  prosthesis  is  the  first  natural  application,  in  principle,  this  approach  should  be  generalizable  to 
other  brain-machine  interfaces  in  the  future. 

2.  KEYWORDS 

Artificial  retina,  Retinal  prosthesis,  Brain-machine  interface.  Brain-computer  interface,  neural  interface,  Vision, 
Artificial  vision 

3.  ACCOMPLISHMENTS 

•  What  were  the  major  goals  of  the  project? 

Task  1:  Natural  Encoding.  Our  goal  was  to  characterize  natural  encoding  performed  by  the  retina  by 
developing  the  most  advanced  models  of  retinal  function.  Natural  encoding  transforms  the  visual  scene  into 
patterns  of  activity  in  ~20  different  types  of  retinal  ganglion  cells  (RGCs),  which  send  distinct  information  about 
the  visual  scene  to  distinct  targets  in  the  brain. 

Task  2:  Natural  Decoding.  Our  goal  was  to  develop  a  model  of  visual  decoding  from  RGC  spike  trains  that  the 
brain  uses  to  produce  perception.  Such  a  decoding  model  provides  our  best  estimate  of  the  visual  images  over 
time  that  the  subject  will  perceive,  given  any  particular  pattern  of  RGC  activity  produced  naturally  or  by  the 
prosthesis.  This  estimate  is  obtained  by  numerical  “inversion”  of  the  encoding  model  to  reconstruct  the  optimal 
perceived  image  from  the  pattern  of  RGC  activity. 

Task  3:  Prosthesis  Encoding.  Our  goal  was  to  optimize  an  emulated  prosthesis  encoder,  which  is  used  to 
transform  the  incoming  image  into  a  pattern  of  electrical  stimulation,  using  the  experimentally  measured 
achievable  set  of  patterns  as  constraints.  This  optimization  will  select,  among  the  set  of  patterns  of  activity  that 
the  electrodes  are  capable  of  producing  in  the  nearby  neurons,  the  pattern  that  results  in  the  most  accurate  visual 
image  possible. 

•  What  was  accomplished  under  these  goals? 

Task  1:  Natural  Encoding.  We  improved  upon  state-of-the  art  models  of  retinal  encoding  by  developing 
machine  learning  approaches  to  characterize  the  retinal  responses  to  naturalistic  stimuli  [4,  5].  We  learned  that 
certain  spatial  and  temporal  nonlinearities,  not  included  in  commonly  used  models,  are  crucial  for  the  improved 
prediction  of  retinal  responses.  In  addition,  we  developed  methods  to  identify  RGC  types  from  their  electrical 
signatures  [7,  8].  This  is  used  in  conjunction  with  models  of  RGC  response  based  on  cell  type  identity,  which  is 
essential  for  applying  our  approaches  to  a  retina  blinded  by  loss  of  photoreceptors. 
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Task  2:  Natural  Decoding.  We  have  accomplished  this  in  two  ways  that  will  soon  merge.  First,  we  have 
developed  linear  reconstruction  of  images  from  actual  recorded  RGC  activity,  and  studied  its  properties  and  how 
to  optimize  it  [10,  1],  Second,  we  used  machine  learning  methods  to  build  on  linear  decoding  with  more  flexible 
nonlinear  computation.  This  required  large  data  sets,  so  we  started  by  doing  this  with  simulated  retinal  responses 
using  encoding  models  (see  Task  1).  This  approach  significantly  outperformed  linear  decoding  [3,  6].  Our  next 
step  will  be  to  merge  these  by  using  nonlinear  decoding  from  recorded  retinal  responses. 

Task  3:  Prosthesis  Encoding.  We  developed  a  greedy  dictionary-based  encoding  approach  that  efficiently  and 
sequentially  selects  electrical  stimuli  that  produce  the  greatest  increment  in  image  reconstruction  [9].  This  is  the 
first  technically  viable  approach  to  this  problem,  is  based  on  the  fundamental  concept  of  producing  a  visual  image 
of  optimal  quality,  and  has  many  extensions  that  we  will  be  working  on,  including:  expansion  of  the  dictionary  by 
multi -electrode  patterns,  nonlinear  reconstruction  (see  Task  2),  and  including  the  statistics  of  retinal  spiking. 

•  What  opportunities  for  training  and  professional  development  has  the  project  provided? 

Nikhil  Parthasarathy  continued  to  NYU  Center  for  Neural  Science  Ph.D.  program.  Orren  Karniol-Tambour  is  now 
applying  to  Ph.D.  programs  in  neuroscience.  Nishal  Shah  advanced  to  Ph.D.  candidacy  during  the  work. 

•  How  were  the  results  disseminated  to  communities  of  interest? 

So  far,  we  have  prepared  a  number  of  conference  presentations  (talks  and  posters)  and  conference  papers  (see 
Products).  We  are  preparing  journal  articles  for  publication  now. 

•  What  do  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals? 

Nothing  to  report. 

4.  IMPACT 

•  What  was  the  impact  on  the  development  of  the  principal  discipline(s)  of  the  project? 

These  developments  were  very  well  received  at  the  major  conferences  in  the  field  (NIPS,  ICLR,  and  The  Eye  and 
the  Chip),  and  have  been  submitted  as  conference  papers,  archive  papers,  and  are  being  worked  on  for  journal 
articles  (see  Products).  Collectively  we  have  substantially  advanced  the  prospects  of  effective  encoding  in  a 
retinal  prosthesis  with  this  approach. 

•  What  was  the  impact  on  other  disciplines? 

Our  approach  to  stimulus  reconstruction  using  artificial  neural  networks,  well  received  at  NIPS,  provides  a  new 
way  to  extend  machine  learning  methods  to  incorporate  spike  trains  of  visual  neurons.  Our  method  for 
determining  cell  types  from  electrical  recordings  has  implications  for  understanding  the  biophysics  of  retinal 
neurons. 

•  What  was  the  impact  on  technology  transfer? 

None  yet.  We  expected  the  techniques  developed  to  be  incorporated  in  the  technology  for  the  device  now  being 
designed  by  the  Stanford  Artificial  Retina  project. 

•  What  was  the  impact  on  society  beyond  science  and  technology? 

Nothing  to  report. 

5.  CHANGES/PROBLEMS 

•  Changes  in  approach  and  reasons  for  change 

Nothing  to  report. 

•  Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them. 

Nothing  to  report. 
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Changes  that  had  a  significant  impact  on  expenditures. 

Nothing  to  report. 

Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards,  and/or  select  agents. 

Nothing  to  report. 

Significant  changes  in  use  or  care  of  human  subjects. 

Nothing  to  report. 

Significant  changes  in  use  or  care  of  vertebrate  animals 

Nothing  to  report. 

Significant  changes  in  use  of  biohazards  and/or  select  agents. 

Nothing  to  report. 

PRODUCTS 

Publications,  conference  papers,  and  presentations 

1 .  Golden  JR,  Erickson-Davis  C,  Cottaris  NP,  Parthasarathy  N,  Rieke  F,  Brainard  DH,  Wandell 

BA,  Chichilnisky  EJ.  (2017)  Simulation  of  visual  perception  and  learning  with  a  retinal  prosthesis.  bioRxiv  doi: 
10.1101/206409 

2.  Shah  NP,  Madugula  S,  Chichilnisky  EJ,  Shlens  J,  Singer  Y.  (2017)  Learning  a  neural 
response  metric  for  retinal  prosthesis.  bioRxiv  doi:  10.1 101/226530 

3.  Parthasarathy  N,  Batty  E,  Falcon  W,  Rutten  T,  Rajpal  M,  Chichilnisky  EJ,  Paninski  L.  (2017) 

Neural  Networks  for  Efficient  Bayesian  Decoding  of  Natural  Images  from  Retinal  Neurons. 

In:  Advances  in  Neural  Information  Processing  Systems,  pp.  6437-48.  (Accepted  as 
Spotlight  Presentation) 

4.  Shah  N,  Brackbill  N,  Rhoades  C,  Tikidji-Hamburyan  A,  Goetz  G,  Litke  A,  Sher  A,  Gupta  V, 

Singer  Y,  Chichilnisky  EJ,  Shlens  J.  (2017)  Learning  nonlinear  models  for  visaed  computation  in  populations  of 
retinal  ganglion  cells.  Presented  at:  The  Computational  and  Systems  Neuroscience  (Cosyne)  2017  Meeting,  2017 
Feb  23-26;  Salt  Lake  City,  UT. 

5.  Batty  E,  Merel  J,  Brackbill  N,  Heitman  A,  Sher  A,  Litke  A,  Chichilnisky  EJ,  Paninski  L. 

Multilayer  Recurrent  Network  Models  of  Primate  Retinal  Ganglion  Cell  Responses. 

Presented  at:  The  5th  International  Conference  on  Learning  Representations  (ICLR),  2017 
Apr  24-26;  Toulon,  FR. 

6.  Parthasarathy  N,  Batty  E,  Falcon  W,  Rutten  T,  Rajpal  M,  Chichilnisky  EJ,  Paninski  L. 

Nonlinear  amortized  Bayesian  decoding  of  natural  scenes  from  retinal  responses. 

Presented  at:  The  Collaborative  Research  in  Computational  Neuroscience  (CRCNS)  2017 
Annual  PI  Meeting,  2017  Jun  14-16;  Providence,  RI. 

7.  Karniol-Tambour  O*,  Goetz  G*,  Grosberg  L,  Rhoades  C,  Brackbill  N,  Shah  NP,  Sher  A, 

Litke  AM,  Chichilnisky  EJ.  Using  deep  learning  and  simple  encoding  models  to  infer  the 
neural  code  for  a  retinal  prosthesis.  Presented  at:  The  France-Stanford  Center  for 
Interdisciplinary  Studies  Workshop  on  Brain-Inspired  Computation,  2017  Fall;  Stanford,  CA. 

(*equal  contribution) 

8.  Goetz  G*,  Karniol-Tambour  O*,  Grosberg  L,  Rhoades  C,  Brackbill  N,  Shah  NP,  Sher  A,  Litke 
AM,  Chichilnisky  EJ.  Inferring  the  neural  code  in  a  blind  retina.  Presented  at:  The  Eye  and 

The  Chip,  World  Research  Congress  on  Artificial  Vision  (TEATC),  2017  Sep  23-26;  Detroit, 


3 

DISTRIBUTION  STATEMENT  A:  Approved  for  public  release. 


Cleared,  88PA,  Case  #2018-0596. 


MI.  (*equal  contribution) 

9.  Shah  N,  Madugula  S,  Grosberg  L,  Mena  G,  Ganesan  K,  Bhaskhar  N,  Hottowy  P,  Dabrowski 
W,  Sher  A,  Litke  AM,  Mitra  S,  Chichilnisky  EJ.  Greedy  dictionary-based  stimulation  for 
optimization  of  epiretinal  prosthesis.  Presented  at:  The  Eye  and  The  Chip,  World  Research 
Congress  on  Artificial  Vision  (TEATC),  2017  Sep  23-26;  Detroit,  MI. 

10.  Brackbill  N,  Parthasarathy  N,  Kamiol-Tambour  O,  Rhoades  C,  Shah  NP,  Goetz  G, 

Tikidji-Hamburyan  A,  Sher  A,  Litke  AM,  Chichilnisky  EJ.  Reconstruction  of  natural  images  from  responses  of 
primate  retinal  ganglion  cells.  Presented  at:  The  European  Retina  Meeting  (ERM),  2017  Oct  5-7;  Paris,  FR. 

•  Website(s)  or  other  Internet  site(s) 

Nothing  to  report. 

•  Technologies  or  techniques 

Nothing  to  report. 

•  Inventions,  patent  applications,  and/or  licenses 

Nothing  to  report. 

•  Other  products 

Nothing  to  report 

7.  PARTICIPANTS  &  OTHER  COLLABORATING  ORGANIZATIONS 


Name: 

Eduardo  J.  Chichilnisky 

Project  Role: 

PI 

Researcher  Identifier  (e.g.  ORCID 
ID): 

0000-0002-5613-0248 

Nearest  person  month  worked: 

6 

Contribution  to  Project: 

Conceived  and  directed  the  project,  oversaw  presentations  and 
publications 

Funding  Support: 

Name: 

Nishal  Shah 

Project  Role: 

Graduate  Student 

Researcher  Identifier  (e.g.  ORCID 
ID): 

Nearest  person  month  worked: 

6 

Contribution  to  Project: 

Development  of  greedy  dictionary-based  algorithm  for  encoding 

Funding  Support: 

Name: 

Orren  Kamiol  Tambour 

Project  Role: 

Programmer 

Researcher  Identifier  (e.g.  ORCID 
ID): 

Nearest  person  month  worked: 

9 

Contribution  to  Project: 

Development  of  methods  to  identify  retinal  ganglion  cell  types  from 
electrical  recordings  and  emulate  responses  with  encoding  models 

4 

DISTRIBUTION  STATEMENT  A:  Approved  for  public  release. 


Cleared,  88PA,  Case  #2018-0596. 


Funding  Support: 

Name: 

Nikhil  Pcirthasarathy 

Project  Role: 

Graduate  Student 

Researcher  Identifier  (e.g.  ORCID 
ID): 

Nearest  person  month  worked: 

8 

Contribution  to  Project: 

Development  of  methods  to  reconstruct  visual  images  from  electrical 
activity  of  retinal  neurons  using  artificial  neural  networks 

Funding  Support: 

Has  there  been  a  change  in  the  active  other  support  of  the  PD/PI(s)  or  senior/key  personnel  since  the  last 
reporting  period? 

Nothing  to  report. 

What  other  organizations  were  involved  as  partners? 

Nothing  to  report. 

SPECIAL  REPORTING  REQUIREMENTS 

Nothing  to  report 


5 

DISTRIBUTION  STATEMENT  A:  Approved  for  public  release. 


Cleared,  88PA,  Case  #2018-0596. 


List  of  Symbols,  Abbreviations  and  Acronyms 


RGC  -  retinal  ganglion  cell 

NIPS  -  Neural  Information  Processing  Systems  Meeting 

ICLR  -  International  Conference  on  Learning  Representations  Meeting 

COSYNE  -  The  Computational  and  Systems  Neuroscience  Meeting 

CRCNS  -  The  Collaborative  Research  in  Computational  Neuroscience  Meeting 

TEATC  -  World  Research  Congress  on  Artificial  Vision 

ERM  -  The  European  Retina  Meeting 

ORCID  ID  -  Open  Researcher  and  Contributor  Identification 


6 

DISTRIBUTION  STATEMENT  A:  Approved  for  public  release. 


Cleared,  88PA,  Case  #2018-0596. 


